{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HTML解析入门及准备URL生成连续技\n",
    "![for humans](https://requests-html.kennethreitz.org/_static/requests-html-logo.png#thumbnail)\n",
    "\n",
    "*  本周主要内容：批量抓取页面基础及技巧\n",
    "*  上周主要内容：HTML解析（parse HTML）及准备URL生成连续技\n",
    "*  20春_Web数据挖掘_week04\n",
    "*  电子讲义设计者：廖汉腾, 许智超\n",
    "<br/>\n",
    "<br/>\n",
    "\n",
    "-----\n",
    "## 复习\n",
    "\n",
    "复习：上周内容，实践\n",
    "\n",
    "* 猎聘PC版 liepin.com 取工作URL参数的牛肉\n",
    "* 如何生成一连串新URL以进一步爬取数据\n",
    "\n",
    "\n",
    "-----\n",
    "## 本周内容及学习目标\n",
    "\n",
    "本周内容聚焦在\n",
    "\n",
    "<mark> 如何有系统的把更多页数据(相同结构)作系统性爬取 </mark>\n",
    "\n",
    "为此，我们需要学习\n",
    "\n",
    "* 翻页：参数字典的拆解\n",
    "  * xpath\n",
    "  * 建构参数模板\n",
    "  * 建构参数字典\n",
    "* 翻页：系统性迭代\n",
    "  * robots.txt\n",
    "  * 频率及时间\n",
    "* 翻页：数据备份与整合\n",
    "  * 储存备份\n",
    "  * 数据整合\n",
    "  \n",
    "### 目标\n",
    "1. 使用 requests-html 爬取并存取网页文字档，查找[requests-html 中文文档](https://cncert.github.io/requests-html-doc-cn/#/)\n",
    "2. 熟悉 [xpath 语法](https://www.w3cschool.cn/xpath/xpath-syntax.html)丶[xpath 节点](https://www.w3cschool.cn/xpath/xpath-nodes.html)\n",
    "3. 使用 [xpath cheatsheet](https://devhints.io/xpath)\n",
    "  * 在 Chrome Inspector 使用\n",
    "  * 在 requests-html (Python) 使用\n",
    "4. 简易使用 [pd.DataFrame](https://www.pypandas.cn/doc/getting_started/dsintro.html#dataframe)\n",
    "5. 参数字典的拆解与迭代\n",
    "6. 翻页数据备份与整合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "/* 本电子讲义使用之CSS */\n",
       "div.code_cell {\n",
       "    background-color: #e5f1fe;\n",
       "}\n",
       "div.cell.selected {\n",
       "    background-color: #effee2;\n",
       "    font-size: 2rem;\n",
       "    line-height: 2.4rem;\n",
       "}\n",
       "div.cell.selected .rendered_html table {\n",
       "    font-size: 2rem !important;\n",
       "    line-height: 2.4rem !important;\n",
       "}\n",
       ".rendered_html pre code {\n",
       "    background-color: #C4E4ff;   \n",
       "    padding: 2px 25px;\n",
       "}\n",
       ".rendered_html pre {\n",
       "    background-color: #99c9ff;\n",
       "}\n",
       "div.code_cell .CodeMirror {\n",
       "    font-size: 2rem !important;\n",
       "    line-height: 2.4rem !important;\n",
       "}\n",
       ".rendered_html img, .rendered_html svg {\n",
       "    max-width: 60%;\n",
       "    height: auto;\n",
       "    float: right;\n",
       "}\n",
       "\n",
       ".rendered_html img[src*=\"#full\"], .rendered_html svg[src*=\"#full\"] {\n",
       "    max-width: 100%;\n",
       "    height: auto;\n",
       "    float: none;\n",
       "}\n",
       "\n",
       ".rendered_html img[src*=\"#thumbnail\"], .rendered_html svg[src*=\"#thumbnail\"] {\n",
       "    max-width: 15%;\n",
       "    height: auto;\n",
       "}\n",
       "\n",
       "/* Gradient transparent - color - transparent */\n",
       "hr {\n",
       "    border: 0;\n",
       "    border-bottom: 1px dashed #ccc;\n",
       "}\n",
       ".emoticon{\n",
       "    font-size: 5rem;\n",
       "    line-height: 4.4rem;\n",
       "    text-align: center;\n",
       "    vertical-align: middle;\n",
       "}\n",
       ".bg-split_apply_comine {\n",
       "    width: 500px;     \n",
       "    height: 300px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -10px -10px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-comine {\n",
       "    width: 175px;\n",
       "    height: 150px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -280px -80px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-apply {\n",
       "    width: 155px;\n",
       "    height: 225px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -160px -30px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-split {\n",
       "    width: 205px;\n",
       "    height: 225px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -10px -30px;\n",
       "    float: right;\n",
       "}\n",
       ".break {\n",
       "                   page-break-after: right; \n",
       "                   width:700px;\n",
       "                   clear:both;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>\n",
    "/* 本电子讲义使用之CSS */\n",
    "div.code_cell {\n",
    "    background-color: #e5f1fe;\n",
    "}\n",
    "div.cell.selected {\n",
    "    background-color: #effee2;\n",
    "    font-size: 2rem;\n",
    "    line-height: 2.4rem;\n",
    "}\n",
    "div.cell.selected .rendered_html table {\n",
    "    font-size: 2rem !important;\n",
    "    line-height: 2.4rem !important;\n",
    "}\n",
    ".rendered_html pre code {\n",
    "    background-color: #C4E4ff;   \n",
    "    padding: 2px 25px;\n",
    "}\n",
    ".rendered_html pre {\n",
    "    background-color: #99c9ff;\n",
    "}\n",
    "div.code_cell .CodeMirror {\n",
    "    font-size: 2rem !important;\n",
    "    line-height: 2.4rem !important;\n",
    "}\n",
    ".rendered_html img, .rendered_html svg {\n",
    "    max-width: 60%;\n",
    "    height: auto;\n",
    "    float: right;\n",
    "}\n",
    "\n",
    ".rendered_html img[src*=\"#full\"], .rendered_html svg[src*=\"#full\"] {\n",
    "    max-width: 100%;\n",
    "    height: auto;\n",
    "    float: none;\n",
    "}\n",
    "\n",
    ".rendered_html img[src*=\"#thumbnail\"], .rendered_html svg[src*=\"#thumbnail\"] {\n",
    "    max-width: 15%;\n",
    "    height: auto;\n",
    "}\n",
    "\n",
    "/* Gradient transparent - color - transparent */\n",
    "hr {\n",
    "    border: 0;\n",
    "    border-bottom: 1px dashed #ccc;\n",
    "}\n",
    ".emoticon{\n",
    "    font-size: 5rem;\n",
    "    line-height: 4.4rem;\n",
    "    text-align: center;\n",
    "    vertical-align: middle;\n",
    "}\n",
    ".bg-split_apply_comine {\n",
    "    width: 500px;     \n",
    "    height: 300px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -10px -10px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-comine {\n",
    "    width: 175px;\n",
    "    height: 150px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -280px -80px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-apply {\n",
    "    width: 155px;\n",
    "    height: 225px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -160px -30px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-split {\n",
    "    width: 205px;\n",
    "    height: 225px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -10px -30px;\n",
    "    float: right;\n",
    "}\n",
    ".break {\n",
    "                   page-break-after: right; \n",
    "                   width:700px;\n",
    "                   clear:both;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 基本模块\n",
    "import pandas as pd\n",
    "from requests_html import HTMLSession"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. 上周整合代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 根据行业搜索公司\n",
    "- 步骤理解：\n",
    "    1. 建构url参数模板(使用urlparse, parse_qs,nunique解析url，并生成搜索参数字典)\n",
    "    2. 建立连接所需模块/参数(引入request_html HTMLSession, 方法:session.get(参数:url,params))\n",
    "    3. 建立连接方法def requests_liepin\n",
    "    4. 使用方法requests_liepin( url, params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>职称</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>公司名称</th>\n",
       "      <th>edu</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>阿里巴巴</th>\n",
       "      <th>学历不限</th>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>小米</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">华为</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>统招本科</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>北京比特大陆科技有限公司</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>小米</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>远东国际融资租赁有限公司</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>明略科技集团</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>网易集团</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>车好多集团</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>黑芝麻智能科技(上海)有限公司</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SenseTime（商汤集团）</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>华为</th>\n",
       "      <th>硕士及以上</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>上海擎创信息技术有限公司</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>达而观信息科技(上海)有限公司</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>小米</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>新东方教育科技集团有限公司</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>宁德时代新能源科技股份有限公司</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>远东控股集团</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>上海擎创信息技术有限公司</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>山东开创集团股份有限公司</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>禹洲地产股份有限公司</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">网易集团</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>统招本科</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>上海擎创信息技术有限公司</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>赛轮集团股份有限公司</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>苏州思必驰</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>阿里巴巴</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>车好多集团</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>新东方教育科技集团有限公司</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NIO蔚来</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CVTE</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>双胞胎</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>百济神州</th>\n",
       "      <th>学历不限</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>百融云创科技股份有限公司</th>\n",
       "      <th>硕士及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>山东荣信集团有限公司</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">岁宝百货</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>平安银行</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>戴维医疗</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>招金矿业</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Baidu</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>宝德投资深圳</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>新城悦控股有限公司</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>新城控股集团住宅开发事业部</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>安徽广信农化股份有限公司</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">方太</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>统招本科</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>旷视科技</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>明略科技集团</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>天际电器</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>欧菲光</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>天士力集团网</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>大疆创新</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>深圳市优必选科技股份有限公司</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>深圳市农产品集团股份有限公司</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>深圳市客路网络科技有限公司</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>猎聘</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>大汉控股集团有限公司</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>数梦工场</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>96 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                       职称\n",
       "公司名称            edu      \n",
       "阿里巴巴            学历不限   20\n",
       "小米              统招本科   14\n",
       "华为              本科及以上  12\n",
       "                统招本科    8\n",
       "北京比特大陆科技有限公司    统招本科    8\n",
       "小米              本科及以上   6\n",
       "远东国际融资租赁有限公司    本科及以上   5\n",
       "明略科技集团          本科及以上   5\n",
       "网易集团            本科及以上   5\n",
       "车好多集团           统招本科    5\n",
       "黑芝麻智能科技(上海)有限公司 本科及以上   5\n",
       "SenseTime（商汤集团） 统招本科    5\n",
       "华为              硕士及以上   4\n",
       "上海擎创信息技术有限公司    本科及以上   4\n",
       "达而观信息科技(上海)有限公司 统招本科    4\n",
       "小米              大专及以上   4\n",
       "新东方教育科技集团有限公司   统招本科    4\n",
       "宁德时代新能源科技股份有限公司 大专及以上   4\n",
       "远东控股集团          统招本科    4\n",
       "上海擎创信息技术有限公司    统招本科    3\n",
       "山东开创集团股份有限公司    大专及以上   3\n",
       "禹洲地产股份有限公司      统招本科    3\n",
       "网易集团            大专及以上   3\n",
       "                统招本科    3\n",
       "上海擎创信息技术有限公司    大专及以上   3\n",
       "赛轮集团股份有限公司      本科及以上   3\n",
       "苏州思必驰           本科及以上   3\n",
       "阿里巴巴            统招本科    2\n",
       "车好多集团           大专及以上   2\n",
       "新东方教育科技集团有限公司   本科及以上   2\n",
       "...                    ..\n",
       "NIO蔚来           统招本科    1\n",
       "CVTE            本科及以上   1\n",
       "双胞胎             大专及以上   1\n",
       "百济神州            学历不限    1\n",
       "百融云创科技股份有限公司    硕士及以上   1\n",
       "山东荣信集团有限公司      本科及以上   1\n",
       "岁宝百货            大专及以上   1\n",
       "                本科及以上   1\n",
       "平安银行            本科及以上   1\n",
       "戴维医疗            统招本科    1\n",
       "招金矿业            统招本科    1\n",
       "Baidu           本科及以上   1\n",
       "宝德投资深圳          大专及以上   1\n",
       "新城悦控股有限公司       大专及以上   1\n",
       "新城控股集团住宅开发事业部   本科及以上   1\n",
       "安徽广信农化股份有限公司    统招本科    1\n",
       "方太              本科及以上   1\n",
       "                统招本科    1\n",
       "旷视科技            本科及以上   1\n",
       "明略科技集团          统招本科    1\n",
       "天际电器            大专及以上   1\n",
       "欧菲光             本科及以上   1\n",
       "天士力集团网          统招本科    1\n",
       "大疆创新            本科及以上   1\n",
       "深圳市优必选科技股份有限公司  统招本科    1\n",
       "深圳市农产品集团股份有限公司  本科及以上   1\n",
       "深圳市客路网络科技有限公司   本科及以上   1\n",
       "猎聘              大专及以上   1\n",
       "大汉控股集团有限公司      统招本科    1\n",
       "数梦工场            本科及以上   1\n",
       "\n",
       "[96 rows x 1 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 上周C-1B-5 建构 参数模板  获取链接、分析链接、生成链接参数模板\n",
    "参数_compTag_用户体验 = {\n",
    "    '中国500强': {'init': ['-1'],'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['155'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'],'d_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, \n",
    "    '2018互联网300强': {'init': ['-1'],'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'],'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['182'], 'ckid': ['58d828c357a8cb19'],'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'],'d_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'],  'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, \n",
    "    '制造业500强': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['186'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, \n",
    "    'AI创新成长50强 ': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['189'], 'ckid': ['58d828c357a8cb19'],  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'],'d_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'],'d_headId': ['6aa779111c1b4ca77cff3648d9dee049']},\n",
    "    '独角兽': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['130'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']},\n",
    "    '上市公司': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['156'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}}\n",
    "\n",
    "# 上周C-1   多个页面准备测试1 中国500强  （建立连接：url，Parmas）\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "session = HTMLSession()\n",
    "payload = 参数_compTag_用户体验['中国500强']\n",
    "#r = session.get( url, params = payload)\n",
    "\n",
    "# r.url\n",
    "\n",
    "# 上周C-2  简化 A-1   单一页面爬+解析\n",
    "session = HTMLSession()\n",
    "\n",
    "def requests_liepin( url, params): #方法：requests_liepin(url,params) ==> r(url,params) ==>session.get( url , params = payload)\n",
    "    r = session.get( url , params = payload)\n",
    "\n",
    "    # 先取特定元素, 精准打击其子后辈\n",
    "    主要元素 = r.html.xpath( '//ul[@class=\"sojob-list\"]/li')\n",
    "\n",
    "    # 作为xpath字典，键为我要抓的牛肉名称，值为xpath\n",
    "    dict_xpaths={ \n",
    "        'text': {\n",
    "            'edu':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]',\n",
    "            '经验':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]/following-sibling::span',\n",
    "            '薪水':    '//div[contains(@class,\"job-info\")]/p/span[@class=\"text-warning\"]', \n",
    "            '时间':    '//div[contains(@class,\"job-info\")]/p/time/@title', \n",
    "            '职称':    '//div[contains(@class,\"job-info\")]/h3/a', \n",
    "            '公司地点': '//div[contains(@class,\"job-info\")]/p/a',\n",
    "            '公司名称': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a', \n",
    "        },\n",
    "        'text_content': {\n",
    "        },\n",
    "        'href': {\n",
    "            '链结':    '//div[contains(@class,\"job-info\")]/h3/a', \n",
    "            '公司URL': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a', \n",
    "        }\n",
    "    }\n",
    "\n",
    "    def get_e_text_content(_xpath_): \n",
    "        # 高级列表推导\n",
    "        暂存结果 = [e.xpath(_xpath_)[0].lxml.text_content() for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_text(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [\"\".join([x.strip() if type(x) is str else x.text.strip() for x in e.xpath(_xpath_)]) for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_href(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [list(e.xpath(_xpath_, first=True).absolute_links)[0] \\\n",
    "                   if len(e.xpath(_xpath_, first=True).absolute_links) >= 1  \\\n",
    "                   else \"\" for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    # 只对主要元素下进行.xpath取值\n",
    "    数据字典 = dict()\n",
    "\n",
    "    数据字典 = {k:get_e_text_content(v) for k,v in dict_xpaths['text_content'].items()} #k:'edu',v:xpath->get_e_text_content(v)->e.xpath(v)[0].lxml.text_content() for e in 主要元素\n",
    "    数据字典.update({k:get_e_text(v) for k,v in dict_xpaths['text'].items()})\n",
    "    数据字典.update({k:get_e_href(v) for k,v in dict_xpaths['href'].items()})\n",
    "\n",
    "    数据 = pd.DataFrame(数据字典)\n",
    "    #数据.to_excel(\"20春_Web数据挖掘_week03_liepin.xlsx\", sheet_name=\"搜查结果\")\n",
    "    return (数据)\n",
    "#print(requests_liepin(url, params = payload)) 单个页面搜查结果\n",
    "\n",
    "#--------------------多个页面:差别:通过遍历参数模板,生成每一行业的链接并查询\n",
    "\n",
    "# 上周C-3   多个页面\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "\n",
    "list_df = list()\n",
    "for k,v in 参数_compTag_用户体验.items(): #k:键（中国500强）v:值（链接参数）\n",
    "    payload = v\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    df = df.assign (热门公司类型 = k)    #df.assign创建或修改列并添加到原数据中,加列，值为公司类型\n",
    "    list_df.append(df) #list_df+df\n",
    "df_all = pd.concat(list_df)\n",
    "#df_all\n",
    "\n",
    "# 上周C-4   输出\n",
    "#df_all.to_excel(\"20春_Web数据挖掘_week03_liepin_各热门公司类型.xlsx\", sheet_name=\"搜查结果\")\n",
    "\n",
    "# 上周C-5 Pandas  基本能力\n",
    "\n",
    "#print (df_all.nunique()) #输出df_all中返回的是唯一值的个数\n",
    "\n",
    "#df_all[['edu']].drop_duplicates() #data中一行元素全部相同时才去除\n",
    "\n",
    "df_all.groupby(['公司名称','edu']).agg({\"职称\":\"count\"}).sort_values(by='职称', ascending=False) #groupby,agg分组统计,sort_values排序"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "-----\n",
    "\n",
    "## 本周实践目标\n",
    "<mark> 如何有系统的把更多页数据(相同结构)作系统性爬取 </mark>[猎聘PC版](https://www.liepin.com/zhaopin/)\n",
    "* 翻页：参数字典的拆解\n",
    "  * xpath解析翻页a/@href\n",
    "  * 建构参数模板\n",
    "  * 建构参数字典\n",
    "* 翻页：系统性迭代\n",
    "  * robots.txt\n",
    "  * 频率及时间\n",
    "* 翻页：数据备份与整合\n",
    "  * 储存备份\n",
    "  * 数据整合"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：参数字典的拆解\n",
    "## xpath解析翻页a/@href"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A-0   单一页面\n",
    "url = \"https://www.liepin.com/zhaopin/?keyword=PRD\"\n",
    "session = HTMLSession()\n",
    "r = session.get( url )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 利用xpath定位翻页文字及链接内容"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/zhaopin/?init=-1&headckid=b7e49e7712e6800f&fromSearchBtn=2&keyword=PRD&ckid=b7e49e7712e6800f°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=476d76cfe8edbee82b52f25802d80cf4&d_curPage=0&d_pageSize=40&d_headId=476d76cfe8edbee82b52f25802d80cf4&curPage=1', '/zhaopin/?init=-1&headckid=b7e49e7712e6800f&fromSearchBtn=2&keyword=PRD&ckid=b7e49e7712e6800f°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=476d76cfe8edbee82b52f25802d80cf4&d_curPage=0&d_pageSize=40&d_headId=476d76cfe8edbee82b52f25802d80cf4&curPage=2', '/zhaopin/?init=-1&headckid=b7e49e7712e6800f&fromSearchBtn=2&keyword=PRD&ckid=b7e49e7712e6800f°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=476d76cfe8edbee82b52f25802d80cf4&d_curPage=0&d_pageSize=40&d_headId=476d76cfe8edbee82b52f25802d80cf4&curPage=3', '/zhaopin/?init=-1&headckid=b7e49e7712e6800f&fromSearchBtn=2&keyword=PRD&ckid=b7e49e7712e6800f°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=476d76cfe8edbee82b52f25802d80cf4&d_curPage=0&d_pageSize=40&d_headId=476d76cfe8edbee82b52f25802d80cf4&curPage=4', '/zhaopin/?init=-1&headckid=b7e49e7712e6800f&fromSearchBtn=2&keyword=PRD&ckid=b7e49e7712e6800f°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=476d76cfe8edbee82b52f25802d80cf4&d_curPage=0&d_pageSize=40&d_headId=476d76cfe8edbee82b52f25802d80cf4&curPage=1', '/zhaopin/?init=-1&headckid=b7e49e7712e6800f&fromSearchBtn=2&keyword=PRD&ckid=b7e49e7712e6800f°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=476d76cfe8edbee82b52f25802d80cf4&d_curPage=0&d_pageSize=40&d_headId=476d76cfe8edbee82b52f25802d80cf4&curPage=9']\n"
     ]
    }
   ],
   "source": [
    "# A-1  xpath 解析翻页a/@href\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a' # 有disabled, current等href是javascript\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'\n",
    "#print (r.html.xpath(xpath_翻页a)) # 物件\n",
    "\n",
    "href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "print (href_列表) #链接的列表\n",
    "\n",
    "文字_列表 = [x.text for x in r.html.xpath(xpath_翻页a)]\n",
    "#print (文字_列表) #链接上的文字\n",
    "\n",
    "href_字典 = {x.text:x.xpath('//@href')[0]  for x in r.html.xpath(xpath_翻页a)}\n",
    "#print (href_字典) #文字：链接（字典）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 观察：\n",
    "此网页是否给出开始丶步进丶及结束的信息，以方便我们完成迭代设置\n",
    "\n",
    "* 老问题 URL太长，用上周的URL+query参数解析与pandas数据框找到异同之处\n",
    "* 老问题 怎麽系统化出URL？用上周的URL+query参数解析与pandas数据框找到异同之处的时候，顺便构建参数字典，至少让以下参数可调\n",
    "  * 搜索关键词：上周keyword\n",
    "  * 页码在哪？\n",
    "* 实践挑战：如何把上周代码模块化为我们所用？\n",
    "\n",
    "-----\n",
    "\n",
    "## 建构参数模板\n",
    "\n",
    "```python\n",
    "\n",
    "# 上周B-1 使用 urllib.parse 解析\n",
    "from urllib.parse import urlparse, parse_qs\n",
    "\n",
    "\n",
    "# 上周B-2 使用 pd.DataFrame进行 unuinque()相异值计量比对 \n",
    "import pandas as pd\n",
    "df = pd.DataFrame([ urlparse(x) for x in 公司数据选择器链结.values()])\n",
    "print(df.nunique())\n",
    "\n",
    "# 上周B-3 针对query 再解析之 \n",
    "#df_qs = pd.DataFrame([ parse_qs(x) for x in df['query'] ])\n",
    "df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "print(df.nunique())\n",
    "\n",
    "# 上周B-4 建构 参数模板 及 字典_compTag\n",
    "def parse_url_qs_for_compTag (url):\n",
    "    six_parts = urlparse(url) \n",
    "    out = parse_qs(six_parts.query)\n",
    "    return (out)\n",
    "\n",
    "# parse_url_qs_for_compTag(list(公司数据选择器链结.values())[0])['compTag']\n",
    "参数模板 = parse_url_qs_for_compTag(list(公司数据选择器链结.values())[0])\n",
    "print(参数模板)\n",
    "# [ parse_url_qs_for_compTag(x)['compTag'] for x in 公司数据选择器链结.values()]\n",
    "[ parse_url_qs_for_compTag(x)['compTag'][0] for x in 公司数据选择器链结.values()]\n",
    "\n",
    "字典_compTag = { k:parse_url_qs_for_compTag(v)['compTag'][0] for k,v in 公司数据选择器链结.items()}\n",
    "print (字典_compTag)\n",
    "\n",
    "# B-5 建构 参数模板  \n",
    "def 参数模板生成(compTag , keyword ):\n",
    "    参数 = 参数模板.copy()\n",
    "    参数['compTag'] = compTag\n",
    "    参数['keyword'] = keyword\n",
    "    return (参数)\n",
    "\n",
    "参数_compTag_用户体验 = { k:参数模板生成(compTag = [v], keyword = ['用户体验']) for k,v in 字典_compTag.items()}\n",
    "print(参数_compTag_用户体验)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 构建参数模板前寻找链接结构：urlparse-->解析url,parse_qs(x)拆分参数\n",
    "    - urlparse-->df\n",
    "    - df['query']-->解析url,parse_qs(x)拆分query中参数-->df_qs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ckid</th>\n",
       "      <th>curPage</th>\n",
       "      <th>d_ckId</th>\n",
       "      <th>d_curPage</th>\n",
       "      <th>d_headId</th>\n",
       "      <th>d_pageSize</th>\n",
       "      <th>d_sfrom</th>\n",
       "      <th>fromSearchBtn</th>\n",
       "      <th>headckid</th>\n",
       "      <th>init</th>\n",
       "      <th>keyword</th>\n",
       "      <th>siTag</th>\n",
       "      <th>curPage_int</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>b7e49e7712e6800f°radeFlag=0</td>\n",
       "      <td>1</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>0</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>40</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>2</td>\n",
       "      <td>b7e49e7712e6800f</td>\n",
       "      <td>-1</td>\n",
       "      <td>PRD</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b7e49e7712e6800f°radeFlag=0</td>\n",
       "      <td>2</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>0</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>40</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>2</td>\n",
       "      <td>b7e49e7712e6800f</td>\n",
       "      <td>-1</td>\n",
       "      <td>PRD</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b7e49e7712e6800f°radeFlag=0</td>\n",
       "      <td>3</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>0</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>40</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>2</td>\n",
       "      <td>b7e49e7712e6800f</td>\n",
       "      <td>-1</td>\n",
       "      <td>PRD</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b7e49e7712e6800f°radeFlag=0</td>\n",
       "      <td>4</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>0</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>40</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>2</td>\n",
       "      <td>b7e49e7712e6800f</td>\n",
       "      <td>-1</td>\n",
       "      <td>PRD</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b7e49e7712e6800f°radeFlag=0</td>\n",
       "      <td>1</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>0</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>40</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>2</td>\n",
       "      <td>b7e49e7712e6800f</td>\n",
       "      <td>-1</td>\n",
       "      <td>PRD</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>b7e49e7712e6800f°radeFlag=0</td>\n",
       "      <td>9</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>0</td>\n",
       "      <td>476d76cfe8edbee82b52f25802d80cf4</td>\n",
       "      <td>40</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>2</td>\n",
       "      <td>b7e49e7712e6800f</td>\n",
       "      <td>-1</td>\n",
       "      <td>PRD</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          ckid curPage                            d_ckId  \\\n",
       "0  b7e49e7712e6800f°radeFlag=0       1  476d76cfe8edbee82b52f25802d80cf4   \n",
       "1  b7e49e7712e6800f°radeFlag=0       2  476d76cfe8edbee82b52f25802d80cf4   \n",
       "2  b7e49e7712e6800f°radeFlag=0       3  476d76cfe8edbee82b52f25802d80cf4   \n",
       "3  b7e49e7712e6800f°radeFlag=0       4  476d76cfe8edbee82b52f25802d80cf4   \n",
       "4  b7e49e7712e6800f°radeFlag=0       1  476d76cfe8edbee82b52f25802d80cf4   \n",
       "5  b7e49e7712e6800f°radeFlag=0       9  476d76cfe8edbee82b52f25802d80cf4   \n",
       "\n",
       "  d_curPage                          d_headId d_pageSize         d_sfrom  \\\n",
       "0         0  476d76cfe8edbee82b52f25802d80cf4         40  search_unknown   \n",
       "1         0  476d76cfe8edbee82b52f25802d80cf4         40  search_unknown   \n",
       "2         0  476d76cfe8edbee82b52f25802d80cf4         40  search_unknown   \n",
       "3         0  476d76cfe8edbee82b52f25802d80cf4         40  search_unknown   \n",
       "4         0  476d76cfe8edbee82b52f25802d80cf4         40  search_unknown   \n",
       "5         0  476d76cfe8edbee82b52f25802d80cf4         40  search_unknown   \n",
       "\n",
       "  fromSearchBtn          headckid init keyword  \\\n",
       "0             2  b7e49e7712e6800f   -1     PRD   \n",
       "1             2  b7e49e7712e6800f   -1     PRD   \n",
       "2             2  b7e49e7712e6800f   -1     PRD   \n",
       "3             2  b7e49e7712e6800f   -1     PRD   \n",
       "4             2  b7e49e7712e6800f   -1     PRD   \n",
       "5             2  b7e49e7712e6800f   -1     PRD   \n",
       "\n",
       "                                           siTag  curPage_int  \n",
       "0  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw            1  \n",
       "1  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw            2  \n",
       "2  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw            3  \n",
       "3  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw            4  \n",
       "4  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw            1  \n",
       "5  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw            9  "
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A-2 建构参数模板：找到关键参数及参数结构\n",
    "\n",
    "# 需要模组库\n",
    "from urllib.parse import urlparse\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "# 总体目标：输入 href_列表, 建构出参数字典\n",
    "\n",
    "# urlparse 解析后丢入数据框\n",
    "df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "\n",
    "#display(df) #df表格\n",
    "#print(df.nunique()) #表格:df中参数的不同值数,发现query\n",
    "#display(df_qs) #df_qs表格\n",
    "#print(df_qs.nunique()) #表格:df_qs中参数的不同值数,发现curPage\n",
    "\n",
    "df_qs.curPage #取出curPage的值观察\n",
    "df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数,加列\"curPage_int\"\n",
    "df_qs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 观察：\n",
    "* query\n",
    "* curPage 5次, 最大值9, 本页不算?\n",
    "\n",
    "-----\n",
    "\n",
    "## 建构参数模板：curPage\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. 建立方法,解析链接.  parse_url_qs_for_curPage,传入参数url,方法:解析url并取出qurey中不同的参数值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'init': ['-1'], 'headckid': ['b7e49e7712e6800f'], 'fromSearchBtn': ['2'], 'keyword': ['PRD'], 'ckid': ['b7e49e7712e6800f°radeFlag=0'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['476d76cfe8edbee82b52f25802d80cf4'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['476d76cfe8edbee82b52f25802d80cf4'], 'curPage': ['1']}\n"
     ]
    }
   ],
   "source": [
    "# A-2 建构参数模板：找到关键参数及参数结构\n",
    "\n",
    "def parse_url_qs_for_curPage (url):\n",
    "    six_parts = urlparse(url) \n",
    "    out = parse_qs(six_parts.query)\n",
    "    return (out)\n",
    "\n",
    "# 取一例做模板，此为例子\n",
    "参数模板 = parse_url_qs_for_curPage(href_列表[0]) #取第一个链接-->方法parse_url_qs_for_curPage-->url解析分段query部分，生成字典\n",
    "print (参数模板)\n",
    "\n",
    "#print (href_字典) #链接及文字的字典"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. 建构参数模板生成器,方法:传入参数模板+自定义curPage+keyword\n",
    "-----\n",
    "5. 使用参数模板生成 --> 形成1-9页的url参数的字典(参数_keyword_用户体验_curPage,key为i页数,value为生成的链接)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "9\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{0: {'init': ['-1'],\n",
       "  'headckid': ['b7e49e7712e6800f'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['b7e49e7712e6800f°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'curPage': [0]},\n",
       " 1: {'init': ['-1'],\n",
       "  'headckid': ['b7e49e7712e6800f'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['b7e49e7712e6800f°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'curPage': [1]},\n",
       " 2: {'init': ['-1'],\n",
       "  'headckid': ['b7e49e7712e6800f'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['b7e49e7712e6800f°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'curPage': [2]},\n",
       " 3: {'init': ['-1'],\n",
       "  'headckid': ['b7e49e7712e6800f'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['b7e49e7712e6800f°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'curPage': [3]},\n",
       " 4: {'init': ['-1'],\n",
       "  'headckid': ['b7e49e7712e6800f'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['b7e49e7712e6800f°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'curPage': [4]},\n",
       " 5: {'init': ['-1'],\n",
       "  'headckid': ['b7e49e7712e6800f'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['b7e49e7712e6800f°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'curPage': [5]},\n",
       " 6: {'init': ['-1'],\n",
       "  'headckid': ['b7e49e7712e6800f'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['b7e49e7712e6800f°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'curPage': [6]},\n",
       " 7: {'init': ['-1'],\n",
       "  'headckid': ['b7e49e7712e6800f'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['b7e49e7712e6800f°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'curPage': [7]},\n",
       " 8: {'init': ['-1'],\n",
       "  'headckid': ['b7e49e7712e6800f'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['b7e49e7712e6800f°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'curPage': [8]},\n",
       " 9: {'init': ['-1'],\n",
       "  'headckid': ['b7e49e7712e6800f'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['b7e49e7712e6800f°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['476d76cfe8edbee82b52f25802d80cf4'],\n",
       "  'curPage': [9]}}"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A-3 建构参数模板生成器：keyword curPage\n",
    "def 参数模板生成(keyword, curPage):\n",
    "    参数 = 参数模板.copy()\n",
    "    参数['curPage'] = curPage\n",
    "    参数['keyword'] = keyword\n",
    "    return (参数)\n",
    "#curPage+keyword+参数模板=参数模板生成\n",
    "\n",
    "参数_keyword_用户体验_curPage = { \n",
    "    i:参数模板生成(curPage = [i], \\\n",
    "                  keyword = ['用户体验']) \n",
    "    for i,v in href_字典.items() } #遍历文字及链接\n",
    "#字典{i=文字（页数），参数生成模板（方法）},这里的i是在页面显示的页数范围\n",
    "   \n",
    " #参数_keyword_用户体验_curPage(字典--> {i:{curPage:i,keywod:'用户体验'}}\n",
    "# print(参数_keyword_用户体验_curPage) # 只生成本页有的额外翻页URL, 并没有推估到&curPage=9,也没有这页\n",
    "\n",
    "print (df_qs.curPage_int.min()) # 最小值只有1\n",
    "print (df_qs.curPage_int.max()) # 最大值只有9  最末页按钮--指向第九页\n",
    "\n",
    "# 应该是 0 (本页)....9(最大值)\n",
    "\n",
    "参数_keyword_用户体验_curPage = { \n",
    "    i:参数模板生成(curPage = [i], \\\n",
    "                  keyword = ['用户体验']) \\\n",
    "    for i in range(0,df_qs.curPage_int.max()+1)\\\n",
    "    } #这里的i是通过使用range及max()+1,从0开始增加1到最大值9\n",
    "参数_keyword_用户体验_curPage #字典：{i:{curPage:i,keywod:'用户体验'+参数模板}} 1-9页的url参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：系统性迭代\n",
    "\n",
    "## 爬亦有道\n",
    "* robots.txt 站长/网站拥有者给搜索引擎的\"道\"\n",
    "* 频率及时间\n",
    "  * 不要爬太快\n",
    "  * 尽量像\"人\"一样礼貌\n",
    "  * time.sleep\n",
    "  \n",
    "```python\n",
    "\n",
    "# 上周C-3   多个页面\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "\n",
    "list_df = list()\n",
    "for k,v in 参数_compTag_用户体验.items():\n",
    "    payload = v\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    df = df.assign (热门公司类型 = k)    \n",
    "    list_df.append(df)\n",
    "\n",
    "df_all = pd.concat(list_df)\n",
    "df_all\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. def requests_liepin建立连接函数,根据xpath取值\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [],
   "source": [
    "# B-1 上周C-2  简化 上上周A-1   单一页面爬+解析\n",
    "session = HTMLSession()\n",
    "\n",
    "def requests_liepin( url, params):\n",
    "    r = session.get( url , params = payload)\n",
    "\n",
    "    # 先取特定元素, 精准打击其子后辈\n",
    "    主要元素 = r.html.xpath( '//ul[@class=\"sojob-list\"]/li')\n",
    "\n",
    "    # 作为xpath字典，键为我要抓的牛肉名称，值为xpath\n",
    "    dict_xpaths={ \n",
    "        'text': {\n",
    "            'edu':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]',\n",
    "            '经验':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]/following-sibling::span',\n",
    "            '薪水':    '//div[contains(@class,\"job-info\")]/p/span[@class=\"text-warning\"]', \n",
    "            '时间':    '//div[contains(@class,\"job-info\")]/p/time/@title', \n",
    "            '职称':    '//div[contains(@class,\"job-info\")]/h3/a', \n",
    "            '公司地点': '//div[contains(@class,\"job-info\")]/p/a',\n",
    "            '公司名称': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a', \n",
    "        },\n",
    "        'text_content': {\n",
    "        },\n",
    "        'href': {\n",
    "            '链结':    '//div[contains(@class,\"job-info\")]/h3/a', \n",
    "            '公司URL': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a', \n",
    "        }\n",
    "    }\n",
    "\n",
    "    def get_e_text_content(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [e.xpath(_xpath_)[0].lxml.text_content() for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_text(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [\"\".join([x.strip() if type(x) is str else x.text.strip() for x in e.xpath(_xpath_)]) for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_href(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [list(e.xpath(_xpath_, first=True).absolute_links)[0] \\\n",
    "                   if len(e.xpath(_xpath_, first=True).absolute_links) >= 1  \\\n",
    "                   else \"\" for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    # 只对主要元素下进行.xpath取值\n",
    "    数据字典 = dict()\n",
    "\n",
    "    数据字典 = {k:get_e_text_content(v) for k,v in dict_xpaths['text_content'].items()}\n",
    "    数据字典.update({k:get_e_text(v) for k,v in dict_xpaths['text'].items()})\n",
    "    数据字典.update({k:get_e_href(v) for k,v in dict_xpaths['href'].items()})\n",
    "\n",
    "    数据 = pd.DataFrame(数据字典)\n",
    "    #数据.to_excel(\"20春_Web数据挖掘_week03_liepin.xlsx\", sheet_name=\"搜查结果\")\n",
    "    return (数据)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 爬亦有道- 不要爬太快\n",
    "time.sleep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7. 加入time.sleep 控制速度 使用time random\n",
    "-----\n",
    "- time.sleep(3+4*random()) random()返回0-1之间的任意数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from random import random\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 4.41 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "time.sleep(3+4*random())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# B-2 多个页面，但放慢脚步 time.sleep\n",
    "\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "\n",
    "list_df = list()\n",
    "for k,v in 参数_keyword_用户体验_curPage.items():\n",
    "    payload = v\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "    df = df.assign (curPage = k)  # 区分  curPage\n",
    "    list_df.append(df)\n",
    "\n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序' #表格前序号\n",
    "# 上周C-4   输出表格\n",
    "#df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页.xlsx\",sheet_name=\"用户体验\")\n",
    "\n",
    "# 预估时间: 5秒*10 =50\n",
    "# 预估数量: 40*10 =400"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function time.sleep>"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## 多个页面+多个关键词\n",
    "time.sleep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "unexpected character after line continuation character (<unknown>, line 23)",
     "output_type": "error",
     "traceback": [
      "Traceback \u001b[1;36m(most recent call last)\u001b[0m:\n",
      "  File \u001b[0;32m\"F:\\Anaconda3\\lib\\site-packages\\IPython\\core\\interactiveshell.py\"\u001b[0m, line \u001b[0;32m3331\u001b[0m, in \u001b[0;35mrun_code\u001b[0m\n    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \u001b[0;32m\"<ipython-input-1-c624549b56df>\"\u001b[0m, line \u001b[0;32m1\u001b[0m, in \u001b[0;35m<module>\u001b[0m\n    get_ipython().run_cell_magic('time', '', '# B-3 多个页面+多个关键词\\nimport time\\nfrom random import random\\n\\nurl = \"https://www.liepin.com/zhaopin/\"\\nxpath_翻页a = \\'//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]\\'\\n\\nkeywords = [\\'用户体验\\',\\'UX\\'] #两个关键词\\nlist_df = list()\\n\\n## 第一页试探有多长的页面\\nfor key in keywords:\\n    payload = 参数模板生成(keyword=[key], curPage=[\\'0\\'])\\n    df = requests_liepin( url, params = payload)\\n    href_列表 = [x.xpath(\\'//@href\\')[0] for x in r.html.xpath(xpath_翻页a)]\\n    df = pd.DataFrame([ urlparse(x) for x in href_列表])\\n    df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df[\\'query\\'] ])\\n    df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数\\n    长度 = df_qs.curPage_int.max()+1\\n    参数_keyword_X_curPage = { \\n        i:参数模板生成(curPage = [i], \\\\\\n                      keyword = [key]) \\\\\\n        for i in range(0,长度)\\\\ #页数\\n        }\\n    #print (参数_keyword_X_curPage)\\n    print (key,长度) \\n    \\n    for k,v in 参数_keyword_X_curPage.items():\\n        payload = v\\n        df = requests_liepin( url, params = payload)\\n        time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\\n        df = df.assign (keyword = key)  # 区分  keyword    \\n        df = df.assign (curPage = k)  # 区分  curPage    \\n        list_df.append(df)\\n        \\ndf_all = pd.concat(list_df).reset_index()\\ndf_all.index.name = \\'序\\'\\n\\n#df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页.xlsx\",sheet_name=\"_\".join(keywords))\\n\\n# 预估时间: 2*5秒*10 =100\\n# 预估数量: 2*40*10 =800\\n')\n",
      "  File \u001b[0;32m\"F:\\Anaconda3\\lib\\site-packages\\IPython\\core\\interactiveshell.py\"\u001b[0m, line \u001b[0;32m2362\u001b[0m, in \u001b[0;35mrun_cell_magic\u001b[0m\n    result = fn(*args, **kwargs)\n",
      "  File \u001b[0;32m\"<decorator-gen-62>\"\u001b[0m, line \u001b[0;32m2\u001b[0m, in \u001b[0;35mtime\u001b[0m\n",
      "  File \u001b[0;32m\"F:\\Anaconda3\\lib\\site-packages\\IPython\\core\\magic.py\"\u001b[0m, line \u001b[0;32m187\u001b[0m, in \u001b[0;35m<lambda>\u001b[0m\n    call = lambda f, *a, **k: f(*a, **k)\n",
      "  File \u001b[0;32m\"F:\\Anaconda3\\lib\\site-packages\\IPython\\core\\magics\\execution.py\"\u001b[0m, line \u001b[0;32m1268\u001b[0m, in \u001b[0;35mtime\u001b[0m\n    expr_ast = self.shell.compile.ast_parse(expr)\n",
      "\u001b[1;36m  File \u001b[1;32m\"F:\\Anaconda3\\lib\\site-packages\\IPython\\core\\compilerop.py\"\u001b[1;36m, line \u001b[1;32m101\u001b[1;36m, in \u001b[1;35mast_parse\u001b[1;36m\u001b[0m\n\u001b[1;33m    return compile(source, filename, symbol, self.flags | PyCF_ONLY_AST, 1)\u001b[0m\n",
      "\u001b[1;36m  File \u001b[1;32m\"<unknown>\"\u001b[1;36m, line \u001b[1;32m23\u001b[0m\n\u001b[1;33m    for i in range(0,长度)\\ #页数\u001b[0m\n\u001b[1;37m                             ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m unexpected character after line continuation character\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# B-3 多个页面+多个关键词\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'\n",
    "\n",
    "keywords = ['用户体验','UX'] #两个关键词\n",
    "list_df = list()\n",
    "\n",
    "## 第一页试探有多长的页面\n",
    "for key in keywords:\n",
    "    payload = 参数模板生成(keyword=[key], curPage=['0'])\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "    df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "    df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "    df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数\n",
    "    长度 = df_qs.curPage_int.max()+1\n",
    "    参数_keyword_X_curPage = { \n",
    "        i:参数模板生成(curPage = [i], \\\n",
    "                      keyword = [key]) \\\n",
    "        for i in range(0,长度)\\ #页数\n",
    "        }\n",
    "    #print (参数_keyword_X_curPage)\n",
    "    print (key,长度) \n",
    "    \n",
    "    for k,v in 参数_keyword_X_curPage.items():\n",
    "        payload = v\n",
    "        df = requests_liepin( url, params = payload)\n",
    "        time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "        df = df.assign (keyword = key)  # 区分  keyword    \n",
    "        df = df.assign (curPage = k)  # 区分  curPage    \n",
    "        list_df.append(df)\n",
    "        \n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "#df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页.xlsx\",sheet_name=\"_\".join(keywords))\n",
    "\n",
    "# 预估时间: 2*5秒*10 =100\n",
    "# 预估数量: 2*40*10 =800"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：数据备份与整合\n",
    "多个页面+多个关键词执行时，若怕中断最好把每一页的df内容备份做中继"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "用户体验 10\n",
      "UX 10\n",
      "产品需求 10\n",
      "PRD 10\n",
      "Wall time: 3min 54s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# C-1 多个页面+多个关键词\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'\n",
    "\n",
    "keywords = ['用户体验','UX','产品需求','PRD']\n",
    "list_df = list()\n",
    "\n",
    "## 第一页试探有多长的页面\n",
    "for key in keywords:\n",
    "    payload = 参数模板生成(keyword=[key], curPage=['0'])\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "    df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "    df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "    df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数\n",
    "    长度 = df_qs.curPage_int.max()+1\n",
    "    参数_keyword_X_curPage = { \n",
    "        i:参数模板生成(curPage = [i], \\\n",
    "                      keyword = [key]) \\\n",
    "        for i in range(0,长度)\\\n",
    "        }\n",
    "    #print (参数_keyword_X_curPage)\n",
    "    print (key,长度)\n",
    "    \n",
    "    for k,v in 参数_keyword_X_curPage.items():\n",
    "        payload = v\n",
    "        df = requests_liepin( url, params = payload)\n",
    "        time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "        ## 备份\n",
    "        df.to_csv(\"20春_Web数据挖掘_week04_liepin_{key}_{k}.tsv\"\\\n",
    "                  .format(key=key, k=k), sep=\"\\t\", encoding=\"utf8\")\n",
    "        \n",
    "        df = df.assign (keyword = key)  # 区分  keyword    \n",
    "        df = df.assign (curPage = k)  # 区分  curPage    \n",
    "        list_df.append(df)\n",
    "        \n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页_4.xlsx\",\\\n",
    "                sheet_name=\"_\".join(keywords))\n",
    "# 预估时间: 4*5秒*10 =200\n",
    "# 预估数量: 4*40*10 =1600"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 本周练习\n",
    "\n",
    "* 开始试验各类参数的调整\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. 尝试修改网站链接\n",
    "- 发现：有些网站的翻页效果不同，实现代码不同"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A-0   单一页面\n",
    "url = \"http://www.nfu.edu.cn/index.php/home/article/search.html?keyword=%E6%96%87%E5%AD%A6%E4%B8%8E%E4%BC%A0%E5%AA%92%E5%AD%A6%E9%99%A2\"\n",
    "session = HTMLSession()\n",
    "r = session.get( url )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/index.php/home/article/search/keyword/%E6%96%87%E5%AD%A6%E4%B8%8E%E4%BC%A0%E5%AA%92%E5%AD%A6%E9%99%A2/p/2.html', '/index.php/home/article/search/keyword/%E6%96%87%E5%AD%A6%E4%B8%8E%E4%BC%A0%E5%AA%92%E5%AD%A6%E9%99%A2/p/3.html', '/index.php/home/article/search/keyword/%E6%96%87%E5%AD%A6%E4%B8%8E%E4%BC%A0%E5%AA%92%E5%AD%A6%E9%99%A2/p/4.html', '/index.php/home/article/search/keyword/%E6%96%87%E5%AD%A6%E4%B8%8E%E4%BC%A0%E5%AA%92%E5%AD%A6%E9%99%A2/p/5.html', '/index.php/home/article/search/keyword/%E6%96%87%E5%AD%A6%E4%B8%8E%E4%BC%A0%E5%AA%92%E5%AD%A6%E9%99%A2/p/6.html', '/index.php/home/article/search/keyword/%E6%96%87%E5%AD%A6%E4%B8%8E%E4%BC%A0%E5%AA%92%E5%AD%A6%E9%99%A2/p/7.html', '/index.php/home/article/search/keyword/%E6%96%87%E5%AD%A6%E4%B8%8E%E4%BC%A0%E5%AA%92%E5%AD%A6%E9%99%A2/p/2.html']\n"
     ]
    }
   ],
   "source": [
    "# A-1  xpath 解析翻页a/@href\n",
    "xpath_翻页a = '//div[@class=\"pages\"]/div/a' # 有disabled, current等href是javascript\n",
    "xpath_翻页a = '//div[@class=\"pages\"]/div/a[starts-with(@href,\"/index.php\")]'\n",
    "#print (r.html.xpath(xpath_翻页a)) # 物件\n",
    "\n",
    "href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "print (href_列表) #链接的列表\n",
    "\n",
    "文字_列表 = [x.text for x in r.html.xpath(xpath_翻页a)]\n",
    "#print (文字_列表) #链接上的文字\n",
    "\n",
    "href_字典 = {x.text:x.xpath('//@href')[0]  for x in r.html.xpath(xpath_翻页a)}\n",
    "#print (href_字典) #文字：链接（字典）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>scheme</th>\n",
       "      <th>netloc</th>\n",
       "      <th>path</th>\n",
       "      <th>params</th>\n",
       "      <th>query</th>\n",
       "      <th>fragment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/index.php/home/article/search/keyword/%E6%96%...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/index.php/home/article/search/keyword/%E6%96%...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/index.php/home/article/search/keyword/%E6%96%...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/index.php/home/article/search/keyword/%E6%96%...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/index.php/home/article/search/keyword/%E6%96%...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/index.php/home/article/search/keyword/%E6%96%...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/index.php/home/article/search/keyword/%E6%96%...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  scheme netloc                                               path params  \\\n",
       "0                /index.php/home/article/search/keyword/%E6%96%...          \n",
       "1                /index.php/home/article/search/keyword/%E6%96%...          \n",
       "2                /index.php/home/article/search/keyword/%E6%96%...          \n",
       "3                /index.php/home/article/search/keyword/%E6%96%...          \n",
       "4                /index.php/home/article/search/keyword/%E6%96%...          \n",
       "5                /index.php/home/article/search/keyword/%E6%96%...          \n",
       "6                /index.php/home/article/search/keyword/%E6%96%...          \n",
       "\n",
       "  query fragment  \n",
       "0                 \n",
       "1                 \n",
       "2                 \n",
       "3                 \n",
       "4                 \n",
       "5                 \n",
       "6                 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# A-2 建构参数模板：找到关键参数及参数结构\n",
    "\n",
    "# 需要模组库\n",
    "from urllib.parse import urlparse\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "# 总体目标：输入 href_列表, 建构出参数字典\n",
    "\n",
    "# urlparse 解析后丢入数据框\n",
    "df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['path'] ])\n",
    "\n",
    "#df[\"path\"]\n",
    "display(df) #df表格\n",
    "#print(df.nunique()) #表格:df中参数的不同值数,发现query\n",
    "#display(df_qs) #df_qs表格\n",
    "#print(df_qs.nunique()) #表格:df_qs中参数的不同值数,发现curPage\n",
    "\n",
    "#df_qs.curPage #取出curPage的值观察\n",
    "#df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数,加列\"curPage_int\"\n",
    "#df_qs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.更改搜索关键词并输出表格"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "新媒体 10\n",
      "网站运营 10\n",
      "界面设计 10\n",
      "产品经理 10\n",
      "Wall time: 3min 55s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# C-1 多个页面+多个关键词\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'\n",
    "\n",
    "keywords = ['新媒体','网站运营','界面设计','产品经理']\n",
    "list_df = list()\n",
    "\n",
    "## 第一页试探有多长的页面\n",
    "for key in keywords:\n",
    "    payload = 参数模板生成(keyword=[key], curPage=['0'])\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "    df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "    df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "    df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数\n",
    "    长度 = df_qs.curPage_int.max()+1\n",
    "    参数_keyword_X_curPage = { \n",
    "        i:参数模板生成(curPage = [i], \\\n",
    "                      keyword = [key]) \\\n",
    "        for i in range(0,长度)\\\n",
    "        }\n",
    "    #print (参数_keyword_X_curPage)\n",
    "    print (key,长度)\n",
    "    \n",
    "    for k,v in 参数_keyword_X_curPage.items():\n",
    "        payload = v\n",
    "        df = requests_liepin( url, params = payload)\n",
    "        time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "        ## 备份\n",
    "        df.to_csv(\"20春_Web数据挖掘_week04_liepin_{key}_{k}.tsv\"\\\n",
    "                  .format(key=key, k=k), sep=\"\\t\", encoding=\"utf8\")\n",
    "        \n",
    "        df = df.assign (keyword = key)  # 区分  keyword    \n",
    "        df = df.assign (curPage = k)  # 区分  curPage    \n",
    "        list_df.append(df)\n",
    "        \n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页_4.xlsx\",\\\n",
    "                sheet_name=\"_\".join(keywords))\n",
    "# 预估时间: 4*5秒*10 =200\n",
    "# 预估数量: 4*40*10 =1600"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "749px",
    "left": "1125.609375px",
    "top": "110px",
    "width": "281.390625px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
